Acoustic scene classification

When interacting with mobile devices we expect relevant information to be presented with a minimum of input effort. What is relevant depends on the context in which we are acting.

If we are requesting a route information while sitting at a busstop we most probably are looking for directions for travel via bus, while at a railway station we most probably are looking for a train connection.

One possibility for a device to identify the context is via geolocation information. But this information may not be available inside buildings. An alternative approach is the analysis of ambient sound. This approach is referred to by the term acoustic scene classification.

Acoustic scene classification (ACS) describes the "capability of a human or anartificial system to understand an audio context, either from an on-linestream or from a recording." (http://www.cs.tut.fi/sgn/arg/dcase2016/documents/workshop/Valenti-DCASE2016workshop.pdf).

This workbook demonstrates how convolutional networks can be used for the classification task.

We will be applying a pre-trained VGG-16 network with a custom classifier applied on log-frequency power spectrograms.

This will provide a trained model which can be used to classify new audio recordings.

The success can be measured by the accuracy of predicting a validation data set. Reaching 70 % accuracy would be a good value for making the results applicable.

Data analysis and preparation

This project uses recordings made available as part of the DCASE (Detection and Classification of Acoustic Scenes and Events) 2019 challenge (http://dcase.community/challenge2019/task-acoustic-scene-classification). The TAU Urban Acoustic Scenes 2019 development dataset contains recordings in 10 different settings (airport, indoor shopping mall, metro station, pedestrian street, stree traffic, tram, bus, metro, park) recorded in 10 cities. Each recording is 10 seconds long. The data files can be downloaded from https://zenodo.org/record/2589280 using data/download.sh.

This workbook assumes that the extracted audio files are in directory data/TAU-urban-acoustic-scenes-2019-development/audio/ relative to this notebook.

In [1]:
import IPython.display as ipd
import librosa, librosa.display
import math
import numpy as np
from collections import OrderedDict
import os
from PIL import Image, ImageDraw, ImageOps
import pandas as pd
import matplotlib.pyplot as plt
import random
from sklearn.metrics import confusion_matrix
from sklearn.preprocessing import MinMaxScaler
import statistics
import torch
from torch import nn, optim
from torchvision import datasets, transforms, models
import warnings
import wave

Let's analyze what the data looks like:

In [2]:
rawDataPath = 'data/TAU-urban-acoustic-scenes-2019-development/audio/'

def filenames2dataFrame(path):
    """Collect metadata of audio files
    
    path   - directory containing audio files
    return - data frame with metadata
    """
    
    data = []
    for filename in os.listdir(path):
        filepath = os.path.join(path, filename)
        head, tail = os.path.split(filename)
        parts = tail.split('-', 2)
        with wave.open(filepath, "rb") as file:
            sr = file.getframerate()
            duration = file.getnframes() / sr
        data.append( {
            'setting' : parts[0],
            'city' : parts[1],
            'recording' : parts[2],
            'duration' : duration,
            'sr' : sr
        })
    return pd.DataFrame(data)

df = filenames2dataFrame(rawDataPath)

print('Durations are in the range {} - {} s.\n'.format(df['duration'].min(), df['duration'].max()))
print('Sample rates are in the range {} - {} Hz.\n'.format(df['sr'].min(), df['sr'].max()))
print('There are a total of {} recordings.\n'.format(df.shape[0]))
print('Number of recordings per category:\n')
print(df.set_index(['setting', 'city', 'duration', 'sr']).count(level='setting'))
print(df.set_index(['setting', 'city', 'duration', 'sr']).count(level='city'))
Durations are in the range 9.999979166666666 - 10.000020833333334 s.

Sample rates are in the range 48000 - 48000 Hz.

There are a total of 14400 recordings.

Number of recordings per category:

                   recording
setting                     
airport                 1440
bus                     1440
metro                   1440
metro_station           1440
park                    1440
public_square           1440
shopping_mall           1440
street_pedestrian       1440
street_traffic          1440
tram                    1440
           recording
city                
barcelona       1421
helsinki        1447
lisbon          1440
london          1444
lyon            1440
milan           1440
paris           1452
prague          1440
stockholm       1456
vienna          1420

This looks benign:

  • For each setting we have the same number of recordings.
  • All recordings are roughly of same duration.
  • All recordings use the same sample rate.

Let's listen to one of the files.

In [3]:
sr = 48000

def readAudioFile(filename, sr=48000):
    """Reads an audio file with default sampling rate 48000Hz
    
    filename - file to be read
    return - numpy.float32
    """
    x, sr = librosa.load(filename, sr=sr)
    return x
In [4]:
x = readAudioFile(rawDataPath + 'street_pedestrian-lyon-1162-44650-a.wav')
print('number of samples {}'.format(x.shape[0]))
ipd.Audio(x, rate=sr)
number of samples 479999
Out[4]:

Spectrograms

To analyze the audio files we can transform them into spectrograms (cf. https://en.wikipedia.org/wiki/Spectrogram). These show the frequency distribution for subsequent short time intervals.

Mel spectrograms

A popular form of spectrograms are Mel spectrograms. The Mel scale is based on what humans perceive as equal pitch differences. The Mel scale defines how the frequency axis is scaled:

$$m = 2595 \log_{10}\left(1 + \frac{f}{700}\right)$$

The result of the scaling is that for high frequencies the scale is proportional to the logarithm of the frequency while low frequency (especially below 700 Hz) are compressed.

For speech analysis this scale is widely used.

In [5]:
def melSpectrogram(x, sr=48000):
    """Draw mel spectrogram
    
    x  - samples
    sr - sample rate
    """
    
    hop_length = 1875 # This gives us 256 time buckets: 1875 = 10 * 48000 / 256
    n_fft = 8192 # This sets the lower frequency cut off to 48000 Hz / 8192 * 2 = 12 Hz
    S = librosa.feature.melspectrogram(x, sr=sr, n_fft=n_fft, hop_length=hop_length)
    logS = librosa.power_to_db(abs(S))
    plt.figure(figsize=(15, 5))
    librosa.display.specshow(logS, sr=sr, hop_length=hop_length, x_axis='time', y_axis='mel')
    plt.colorbar(format='%+2.0f dB')
    plt.title('Mel power spectrum')
    
melSpectrogram(x)
  • Most of the audio power is concentrated on low frequencies.
  • A lot of the structure that we can see is in the low frequencies.
  • High frequencies seem to be filtered out.

Ambiant sound can contain a lot of low frequency sounds, e.g.

  • Automobile motors run at 600 - 6000 rpm. With four cylinders that results in a 40 - 400 Hz exhaust frequency.
  • Ventilators typically run at 750 - 3000 rpm.

These are the frequencies that are compressed by the Mel scale.

When the running speed of machines is changed this will move much of the sound spectrum by the same factor. While the Mel scale distorts this shift for low frequencies the spectrum would be simply translated along the frequency axis on a pure logarithimic scale by the same distance.

So using a logarithmic scale for the the analysis seems more appropriate.

Let's create a log-frequency power spectrogram (also referred to as constant-Q power spectrogram).

Log-frequency spectrograms

In [6]:
def logFrequencySpectrogram(x, sr=48000):

    hop_length = 1024 # must be a multiple of 2^number_of_octaves
    n_bins = 256 # number of bins
    bins_per_octave = 24 # a quarter note
    fmin = 10 # 10 Hz
    fmax = fmin * math.pow(2., n_bins / bins_per_octave)
    C = librosa.cqt(x, sr=sr, fmin = 10, hop_length=hop_length, n_bins= n_bins,
                    bins_per_octave = bins_per_octave)
    logC = librosa.power_to_db(np.abs(C))
    return logC

def drawLogFrequencySpectrogram(x, sr=48000):
    logC = logFrequencySpectrogram(x, sr)
    plt.figure(figsize=(15, 5))
    librosa.display.specshow(logC, sr=sr, x_axis='time', y_axis='cqt_hz', fmin = 10, bins_per_octave = 24)
    plt.colorbar(format='%+2.0f dB')
    plt.title('Constant-Q power spectrum')
    plt.show()
In [7]:
drawLogFrequencySpectrogram(x)
  • With the log-frequency spectrogram the structure of the low frequency sound is clearly visible.
  • The audio power is more distributed over the frequency intervals.
  • Yet high frequencies are still underrepresented.

Pre-emphasis

The high frequencies can be emphasized using a filter.

In [8]:
def preEmphasis(x, alpha=.97):
    """emphasise high frequencies
    
    This filter mixes the signal with its first derivative:
    
    y[n] = (x[n] - alpha * x[n-1]) / (1 - alpha)

    Reference:
    Haytham Fayek,
    "Speech Processing for Machine Learning: Filter banks, Mel-Frequency Cepstral
    Coefficients (MFCCs) and What's In-Between",
    https://haythamfayek.com/2016/04/21/speech-processing-for-machine-learning.html
    
    The implementation by Haytham Fayek lacks a good first value. This
    implementation uses x[-1] = 2 * x[0] - x[1].
    Further to keep the amplitudes for low frequencies unchanged the output signal
    is divided by (1 - alpha).

    x - original signal
    alpha - proportion of the output signal that comes from the first derivative
    """
    return np.append((1 - 2 * alpha) * x[0] + alpha * x[1],
                     x[1:] - alpha * x[:-1]) / (1 - alpha)

p = preEmphasis(x, .97)
ipd.Audio(p, rate=sr)
Out[8]:
In [9]:
drawLogFrequencySpectrogram(p)

The spectogram can easily be converted to a black and white image. This is the input format that we will use for training the neural network.

In [10]:
def bwLogFrequencySpectrogram(x, sr=48000):
    """Create a log-requency spectrogram as an image
    
    x      - samples
    return - spectrogram
    """
    logC = logFrequencySpectrogram(p);
    scaler = MinMaxScaler(feature_range=(0, 255))
    logC = scaler.fit_transform(logC)
    img = Image.fromarray(logC)
    img = img.transpose(Image.FLIP_TOP_BOTTOM)
    return img

def drawImage(img):
    plt.figure(figsize=(12, 7))
    plt.imshow(img, cmap=plt.cm.gray)
        

drawImage(bwLogFrequencySpectrogram(p))

The accompanying script convert.py can be used to convert all recordings to spectrograms

data/TAU-urban-acoustic-scenes-2019-development/audio/ data/spectrograms/

The script saves the data in directory structure like

|- airport
||- barcelona
||- helsinki
|...
|- bus
||- barcelona
||- helsinki
...

In the following it is assumed that the spectrograms are available in directory data/spectrograms relative to this notebook.

In [11]:
# Let's check
img = Image.open('data/spectrograms/street_pedestrian/lyon/1162-44650-a.png')
drawImage(img)
# You should see the same image as above

Modelling

Data augmentation

The number of recordings per category are rather small. To avoid overfitting data augmentation should be applied.

For image data a large variety of transformtions can be used for augmentation. These include for instance random resized cropping, rotations, and flipping (see for instance https://github.com/aleju/imgaug).

Not all make sense for spectrograms, eg. rotations. Reasonable transformations are:

In [12]:
class RandomAudioTransform:
    """Randomly transform short time spectrogram

    The spectrogram is transformed randomly in three ways:

    * The spectrogram frequencies are transposed randomly.
    * The tempo is randomly changed.
    * A random sampling window in time is chosen.

    size             - target image size of the spectrogram
    octaves          - number of octaves by which to shift spectrogram
    bins_per_octaves - number of image pixels per octave
    dilation         - maximum relative increment of the tempo
    sample_size      - size of the time window in relation to the whole
                       spectrogram
    random           - True: use random values, False: replace random values by
                       fixed values
    """

    def __init__(self, size=224, octaves=.5, bins_per_octave=24, dilation=0.25,
                 sample_size=.5, random=True):
        """Construct new RandomAudioTransform
        """

        self.size = size
        self.octaves = octaves
        self.bins_per_octave = bins_per_octave
        self.dilation = dilation
        self.sample_size = sample_size
        self.random = random

    def rand(self):
        """Generate random number from interval [0., 1.[
        """

        if self.random:
            return random.random()
        return .5

    def __call__(self, img):
        """Transform a spectrogram provided as image

        img    - image to transform
        Return - transformed image
        """

        # Stretch the time axis according sample size and time dilation
        width = int((1. + self.dilation * self.rand())
                    * self.size / self.sample_size)
        img = img.resize(size=[width, img.size[1]], resample=Image.BICUBIC)
        # Take sample from image
        alpha = self.octaves * self.bins_per_octave / (img.size[0] - self.size)
        center = [self.rand(), (1 - alpha) * .5 + alpha * self.rand()]
        img = ImageOps.fit(img, size=[self.size, self.size],
                           method=Image.BICUBIC, centering=center)
        return img
In [13]:
# By default a random 5 second sample of the spectrogram shifted by up to .5 octavs is created.
# Just reexecute this cell to see the random changes.
transform = RandomAudioTransform()
img = Image.open('data/spectrograms/street_pedestrian/lyon/1162-44650-a.png')
drawImage(transform(img))
pass

In "SpecAugment: A Simple Augmentation Method for Automatic Speech Recognition", 2019, Zoph et.al suggest masking frequency bands for augmentation (https://ai.google/research/pubs/pub48482/, https://arxiv.org/pdf/1904.08779.pdf).

In [14]:
class FrequencyMask:
    """Randomly mask frequency band in short time spectrogram

    The spectrogram is transformed randomly in three ways:

    * The spectrogram frequencies are transposed randomly.
    * The speed is randomly changed.
    * A random sampling window in time is chosen.

    max_width        - maximum portion of all frequencies that will be masked
    """

    def __init__(self, max_width = .2):
        """Construct new FrequencyMask
        """

        self.max_width = max_width

    def __call__(self, img):
        """Transform a spectrogram provided as as image

        img    - image to transform
        Return - transformed image
        """
        width, height = img.size
        mask_height = height * self.max_width * random.random()
        ymin = math.floor((height - mask_height) * random.random())
        ymax = math.floor(mask_height + ymin)
        draw = ImageDraw.Draw(img)
        draw.rectangle(((0, ymin), (width - 1, ymax)), fill=(127, 127, 127))
        return img
In [35]:
# A random frequency band is masked.
# Just reexecute this cell to see the random changes.
transform = FrequencyMask()
img = Image.open('data/spectrograms/street_pedestrian/lyon/1162-44650-a.png')
drawImage(transform(img))
pass

Train, validation, and test data sets

In machine learning we should use separate data sets for training, validation and testing. Accompanying script split.py is provided for executing the split with using a 80:20:20 ratio on each setting-city combination:

python ./split.py data/spectrograms data/splitted

In the following it is assumed that the splitted data set is available in data/splitted.

In [16]:
# Let's check (assuming Linux as operating system). The result should be close to 8640, 2880, 2880
print("train: {}".format(os.popen("find data/splitted/train -name '*.png' | wc -l").read()))
print("valid: {}".format(os.popen("find data/splitted/valid -name '*.png' | wc -l").read()))
print("test: {}".format(os.popen("find data/splitted/test -name '*.png' | wc -l").read()))
train: 8601

valid: 2901

test: 2898

Neural network training

Using a pre-trained network as starting point very much simplifies finding good solutions. One example of a pre-tained network for image recognition is VGG16 (Karen Simonyan, Andrew Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition", 2014, https://arxiv.org/abs/1409.1556) delivered with the torchvision package.

I have tested both VGG16 and VGG19 in this project but found no benefit in using VGG19 which has some additional layers.

Transforms

In [17]:
def create_transforms(octaves=.5, bins_per_octave=24, dilation=0.25,
                 sample_size=.5, max_width=.2):
    """Create the train and test tranforms

    octaves          - number of octaves by which to shift spectrogram
    bins_per_octaves - number of image pixels per octave
    dilation         - maximum relative increment of the tempo
    sample_size      - size of the time window in relation to the whole
                       spectrogram
    max_width        - maximum portion of all frequencies that will be masked
    return           - training and testing/validation transforms
    """

    # This normalization has been used for pre-training the VGG16 model.
    transform_norm = transforms.Normalize(
        [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    train_transforms = transforms.Compose([RandomAudioTransform(
        octaves=octaves, bins_per_octave=bins_per_octave, dilation=dilation,
        sample_size=sample_size, random=True),
                                           FrequencyMask(max_width),
                                           transforms.ToTensor(),
                                           transform_norm])

    test_transforms = transforms.Compose([RandomAudioTransform(
        random=False),
                                          transforms.ToTensor(),
                                          transform_norm])

    return train_transforms, test_transforms
    

Data loaders

In [18]:
data_dir = 'data/splitted'

def create_data_loaders(train_transforms, test_transforms, data_dir='data/splitted'):
    """Data loaders
    
    data_dir - directory with data
    train_transforms - transforms for training
    test_transforms - transforms for validation and testing
    """
    
    train_dir = os.path.join(data_dir, 'train')
    test_dir = os.path.join(data_dir, 'test')
    valid_dir = os.path.join(data_dir, 'valid')

    train_data = datasets.ImageFolder(train_dir, transform=train_transforms)
    valid_data = datasets.ImageFolder(valid_dir, transform=test_transforms)
    test_data = datasets.ImageFolder(test_dir, transform=test_transforms)

    class_to_idx = test_data.class_to_idx

    # The batch sizes below fit into the 6GB of an Nvidia GTX 1060.
    train_loader = torch.utils.data.DataLoader(
    train_data, batch_size=32, shuffle=True)
    valid_loader = torch.utils.data.DataLoader(
    valid_data, batch_size=32, shuffle=True)
    test_loader = torch.utils.data.DataLoader(
    test_data, batch_size=32, shuffle=True)
    
    return train_loader, test_loader, valid_loader, class_to_idx

Model

In [19]:
def get_in_features(model):
    """Get the number of in_features of the classifier
    """
    in_features = 0
    for module in model.classifier.modules():
        try:
            in_features = module.in_features
            break
        except AttributeError:
            pass
    return in_features

def create_classifier(model, out_features, hidden_units=512):
    """Create the classifier
    """
    classifier = nn.Sequential(OrderedDict([
        ('fc1', nn.Linear(get_in_features(model), hidden_units)),
        ('relu1', nn.ReLU(inplace=True)),
        ('drop1', nn.Dropout(.5)),
        ('fc2', nn.Linear(hidden_units, hidden_units)),
        ('relu2', nn.ReLU(inplace=True)),
        ('drop2', nn.Dropout(.5)),
        ('fc3', nn.Linear(hidden_units, out_features)),
        ('output', nn.LogSoftmax(dim=1))
    ]))
    model.classifier = classifier
In [20]:
def load_model(out_features, hidden_units, classifier_only = True):
    """ Load the pretrained model
    
    out_features    - number of features to predict
    classifier_only - change classifier parameters only
    """
    
    method_to_call = getattr(models, 'vgg16')
    model = method_to_call(pretrained=True)

    if classifier_only:
        # Do not update model parameters
        for param in model.parameters():
            param.requires_grad = False

    # Add our own classifier
    create_classifier(model, out_features=out_features,
                      hidden_units=hidden_units)
    
    return model
In [21]:
def check_accuracy(model, test_loader, device='cuda', mute=True, print_every=40):
    """Check the accuracy of the model"""
    correct = 0
    total = 0
    steps = 0
    expected = []
    actual = []

    model.to(device)
    model.eval()

    with torch.no_grad():
        for inputs, labels in test_loader:
            steps += 1
            inputs, labels = inputs.to(device), labels.to(device)
            outputs = model(inputs)
            _, predicted = torch.max(outputs.data, 1)

            for label in labels.tolist():
                expected.append(label)
            for prediction in predicted.tolist():
                actual.append(prediction)

            total += labels.size(0)
            correct += (predicted == labels).sum().item()

            if not mute:
                print('-', end='', flush=True)
                if steps % print_every == 0:
                    print()
    accuracy = 100 * correct / total    
    if not mute:
        print('\nAccuracy: {:.4f} %'.format(accuracy))
        print('Confusion matrix')
        print(confusion_matrix(expected, actual))
    return accuracy
In [22]:
def save_checkpoint(path, model, optimizer, class_to_idx):
    """Save a checkpoint"""
    model.class_to_idx = class_to_idx
    warnings.filterwarnings("ignore", category=UserWarning)
    torch.save({'model': model,
                'optimizer': optimizer}, path)
    warnings.filterwarnings("default", category=UserWarning)
In [23]:
def do_deep_learning(model, criterion, optimizer, train_loader, valid_loader,
                     class_to_idx, epochs=1, device='cuda', print_every=40,
                     mute=True):
    """Execute training steps
    
    Training is discontinued if there is no improvement for 10 epochs.
    """

    model.to(device)
    model.train()
    
    best_accuracy = 0.0
    worse_count = 0
    
    for epoch in range(epochs):
        running_loss = 0
        steps = 0
        for inputs, labels in train_loader:
            steps += 1

            inputs, labels = inputs.to(device), labels.to(device)

            optimizer.zero_grad()

            # Forward and backward passes
            outputs = model.forward(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            if not mute:
                running_loss += loss.item()
                print('.', end='', flush=True)
                if steps % print_every == 0:
                    print("Epoch: {}/{}, ".format(epoch + 1, epochs),
                          "Loss: {:.4f}".format(running_loss/print_every))
                running_loss = 0

        if not mute:
            while steps % print_every != 0:
                print(' ', end='')
                steps += 1
            print("Epoch {} completed".format(epoch + 1))
        accuracy = check_accuracy(model, valid_loader, mute=mute)
        if mute:
            print("*", end='')
        
        if accuracy > best_accuracy:
            best_accuracy = accuracy
            worse_count = 0
            # To avoid disk wear the checkpoint is saved in the
            # /tmp directory which hopefully is a RAM disk.
            save_checkpoint('/tmp/checkpoint.pt', model, optimizer, class_to_idx)
        else:
            worse_count += 1
            if worse_count == 10:
                if not mute:
                    print("No improvement for 10 epochs")
                break

    print()
    return best_accuracy
In [24]:
def run_training(octaves=.5, bins_per_octave=24, dilation=0.25,
                 sample_size=.5, max_width = .2,
                 hidden_units=512, epochs=200, mute=True):
    """ Run training for given hyper parameters

    octaves          - number of octaves by which to shift spectrogram
    bins_per_octaves - number of image pixels per octave
    dilation         - maximum relative increment of the tempo
    sample_size      - size of the time window in relation to the whole
                       spectrogram
    max_width        - maximum portion of all frequencies that will be masked
    return           - training and testing/validation transforms
    """
    
    train_transforms, test_transforms = \
        create_transforms(octaves=octaves, bins_per_octave=bins_per_octave, dilation=dilation,
                          sample_size=sample_size, max_width = max_width)
    train_loader, test_loader, valid_loader, class_to_idx = \
        create_data_loaders(train_transforms, test_transforms)
    model = load_model(len(class_to_idx), hidden_units)
        
    optimizer = optim.Adam(model.classifier.parameters(), lr=.001)
    criterion = nn.NLLLoss()
    accuracy = do_deep_learning(model, criterion, optimizer, epochs=epochs,
                                train_loader=train_loader, valid_loader=valid_loader,
                                class_to_idx=class_to_idx, mute=mute)
    
    return accuracy, model, test_loader, test_transforms, class_to_idx

# To make this reproducible let'use fixed random seeds
torch.manual_seed(8259)
random.seed(6745)
accuracy, model, test_loader, test_transforms, class_to_idx = run_training(epochs=200,mute=False)
print('Achieved accuracy: {}'.format(accuracy))
........................................Epoch: 1/200,  Loss: 0.0516
........................................Epoch: 1/200,  Loss: 0.0505
........................................Epoch: 1/200,  Loss: 0.0451
........................................Epoch: 1/200,  Loss: 0.0483
........................................Epoch: 1/200,  Loss: 0.0429
........................................Epoch: 1/200,  Loss: 0.0434
.............................           Epoch 1 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 48.3971 %
Confusion matrix
[[213   0   0   0   5   9  34  22   4   4]
 [  1 253   1   2   1   1   0   1   4  26]
 [ 11  35  27  18  16   6  17   2  21 138]
 [ 62  13  11  18  10  18  55   4  51  48]
 [  6   0   0   0 210  27   0   1  27  19]
 [ 55   1   3   2  31  68  32  34  62   2]
 [120   3   1   2   1   4 141  12   6   0]
 [126   3   0   2  10  34  30  60  17   7]
 [  2   3   0   0  25  17   3   7 229   4]
 [  8  53   8   5  22   3   3   0   3 185]]
........................................Epoch: 2/200,  Loss: 0.0382
........................................Epoch: 2/200,  Loss: 0.0295
........................................Epoch: 2/200,  Loss: 0.0339
........................................Epoch: 2/200,  Loss: 0.0362
........................................Epoch: 2/200,  Loss: 0.0340
........................................Epoch: 2/200,  Loss: 0.0297
.............................           Epoch 2 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 56.8769 %
Confusion matrix
[[176   1   3   1   5  13  60  27   2   3]
 [  0 250   5   7   2   1   0   0   3  22]
 [  8  29 129   7  15   4  16   2   5  76]
 [ 48  11  52  55  15   7  56   8  20  18]
 [  3   0   5   0 247  15   0   4  13   3]
 [ 27   0   1   5  36  83  32  49  56   1]
 [ 56   3   1   4   0   3 201  20   2   0]
 [ 66   2   1   0  11  32  48 112  16   1]
 [  0   5   2   3  21  20   3   7 229   0]
 [  7  27  51   6  18   4   5   0   4 168]]
........................................Epoch: 3/200,  Loss: 0.0412
........................................Epoch: 3/200,  Loss: 0.0266
........................................Epoch: 3/200,  Loss: 0.0396
........................................Epoch: 3/200,  Loss: 0.0388
........................................Epoch: 3/200,  Loss: 0.0299
........................................Epoch: 3/200,  Loss: 0.0278
.............................           Epoch 3 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 57.4974 %
Confusion matrix
[[246   0   7   1   1   8  15  10   2   1]
 [  2 203  42   8   0   0   0   2   2  31]
 [ 15   7 185  17   2   7   8   4   3  43]
 [ 60   3  50  89   4   6  43   7  14  14]
 [  7   0   7   1 196  46   0   5  22   6]
 [ 63   0   2   7   7 101  15  46  48   1]
 [ 98   1   3   6   0   2 160  20   0   0]
 [102   0   4   5   0  38  20 108  11   1]
 [  5   2   2   6  15  21   0  10 227   2]
 [ 10   4  95  14   7   4   0   1   2 153]]
........................................Epoch: 4/200,  Loss: 0.0344
........................................Epoch: 4/200,  Loss: 0.0326
........................................Epoch: 4/200,  Loss: 0.0291
........................................Epoch: 4/200,  Loss: 0.0296
........................................Epoch: 4/200,  Loss: 0.0248
........................................Epoch: 4/200,  Loss: 0.0348
.............................           Epoch 4 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 57.2216 %
Confusion matrix
[[126   0   1   4   6  43   6 101   1   3]
 [  0 199  13  11  13   1   4   3   2  44]
 [  6   8 124  33  13  18   2   8   5  74]
 [ 21   3  20 111  15  16  30  42  18  14]
 [  0   0   0   0 258  19   0   6   6   1]
 [  7   0   1   3  30 136   9  66  37   1]
 [ 58   0   1  14   1  16 121  77   2   0]
 [ 11   0   1   3   8  65   5 191   5   0]
 [  0   2   0   5  25  34   0  13 209   2]
 [  1   2  43  17  24   5   0   9   4 185]]
........................................Epoch: 5/200,  Loss: 0.0321
........................................Epoch: 5/200,  Loss: 0.0266
........................................Epoch: 5/200,  Loss: 0.0252
........................................Epoch: 5/200,  Loss: 0.0378
........................................Epoch: 5/200,  Loss: 0.0320
........................................Epoch: 5/200,  Loss: 0.0279
.............................           Epoch 5 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 61.0134 %
Confusion matrix
[[200   1   2   3   3  16  48  16   1   1]
 [  0 258  14   4   0   1   0   1   2  10]
 [ 11  33 104  74   8  10   9   5   1  36]
 [ 31  10  11 143   8   6  50  17   6   8]
 [  4   1   0   0 239  27   0  15   3   1]
 [ 36   1   1   7  22 128  21  48  26   0]
 [ 48   1   0  11   0   5 195  29   1   0]
 [ 53   1   1   8   3  43  28 149   3   0]
 [  2   4   0   5  26  40   1  11 200   1]
 [  2  26  51  30  17   1   5   3   1 154]]
........................................Epoch: 6/200,  Loss: 0.0233
........................................Epoch: 6/200,  Loss: 0.0298
........................................Epoch: 6/200,  Loss: 0.0289
........................................Epoch: 6/200,  Loss: 0.0297
........................................Epoch: 6/200,  Loss: 0.0247
........................................Epoch: 6/200,  Loss: 0.0300
.............................           Epoch 6 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 60.2206 %
Confusion matrix
[[176   1   1   6   6   9  23  63   1   5]
 [  0 259   2   3   0   0   0   2   1  23]
 [  8  29  64  31   3   1   4   8   0 143]
 [ 24   9   4 126   6   2  34  33   4  48]
 [  2   0   0   1 257  11   0   5   4  10]
 [ 25   2   2  16  45  78  17  84  19   2]
 [ 55   2   2  17   0   0 167  47   0   0]
 [ 29   1   0  13   9  12  19 200   3   3]
 [  2   5   0   9  36  26   1  23 187   1]
 [  2  19   6  20   6   0   0   2   2 233]]
........................................Epoch: 7/200,  Loss: 0.0283
........................................Epoch: 7/200,  Loss: 0.0239
........................................Epoch: 7/200,  Loss: 0.0240
........................................Epoch: 7/200,  Loss: 0.0270
........................................Epoch: 7/200,  Loss: 0.0199
........................................Epoch: 7/200,  Loss: 0.0263
.............................           Epoch 7 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 61.8407 %
Confusion matrix
[[226   1   7   4   8   9   1  30   2   3]
 [  0 248   6   0   0   0   0   1   0  35]
 [  3  24 138   3   3   1   0   4   6 109]
 [ 39  14  41  89   6   3  19  17  12  50]
 [  5   1   0   1 255   3   0   2  19   4]
 [ 31   4   5   3  44  90   7  63  37   6]
 [ 93   4   6  14   0   3 121  47   2   0]
 [ 54   1   3   4  14  24   3 175   5   6]
 [  1   6   0   3  20  13   0  12 231   4]
 [  4  13  31   8   7   1   0   3   2 221]]
........................................Epoch: 8/200,  Loss: 0.0275
........................................Epoch: 8/200,  Loss: 0.0220
........................................Epoch: 8/200,  Loss: 0.0229
........................................Epoch: 8/200,  Loss: 0.0190
........................................Epoch: 8/200,  Loss: 0.0305
........................................Epoch: 8/200,  Loss: 0.0238
.............................           Epoch 8 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 64.4261 %
Confusion matrix
[[189   0   0   4   7  14  34  37   2   4]
 [  0 261   5   4   6   0   1   1   2  10]
 [ 10  21 155  13  15   6   2   7   1  61]
 [ 36   6  26 122  11   6  42  15   8  18]
 [  1   0   0   1 252  24   0   6   5   1]
 [ 22   1   2   8  22 134  15  57  28   1]
 [ 58   3   1  12   0   2 183  30   1   0]
 [ 45   0   2  11   9  24  19 175   4   0]
 [  0   4   0   6  25  26   1  12 216   0]
 [  4  20  39  17  21   2   2   1   2 182]]
........................................Epoch: 9/200,  Loss: 0.0231
........................................Epoch: 9/200,  Loss: 0.0236
........................................Epoch: 9/200,  Loss: 0.0198
........................................Epoch: 9/200,  Loss: 0.0250
........................................Epoch: 9/200,  Loss: 0.0335
........................................Epoch: 9/200,  Loss: 0.0336
.............................           Epoch 9 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 63.3230 %
Confusion matrix
[[134   0   1   7   4   8  63  69   1   4]
 [  0 238  11   8   3   0   1   3   3  23]
 [  7  12 155  35   9   3   3   8   1  58]
 [ 20   4  25 144   6   2  47  22   3  17]
 [  3   0   0   2 262  15   0   5   2   1]
 [ 17   0   1  15  33 108  19  80  16   1]
 [ 19   1   3  14   0   1 199  53   0   0]
 [ 18   0   1  12   8  20  27 201   2   0]
 [  1   3   0  12  24  31   1  17 201   0]
 [  2   7  42  26   9   1   1   6   1 195]]
........................................Epoch: 10/200,  Loss: 0.0290
........................................Epoch: 10/200,  Loss: 0.0175
........................................Epoch: 10/200,  Loss: 0.0299
........................................Epoch: 10/200,  Loss: 0.0311
........................................Epoch: 10/200,  Loss: 0.0256
........................................Epoch: 10/200,  Loss: 0.0263
.............................           Epoch 10 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 63.5988 %
Confusion matrix
[[221   1   4   4   1  17  22  19   2   0]
 [  0 275   4   2   0   1   0   1   2   5]
 [ 13  28 191  18   0   5   6   7   5  18]
 [ 36  12  30 123   3   6  45  20   9   6]
 [  4   1   2   4 196  41   0  12  29   1]
 [ 26   3   1   5   7 129  16  65  38   0]
 [ 60   1   1   9   0   6 169  42   2   0]
 [ 45   1   1   6   0  42  17 174   3   0]
 [  0   4   0   5   8  20   1  11 241   0]
 [  4  36  85  19   3   4   1   7   5 126]]
........................................Epoch: 11/200,  Loss: 0.0202
........................................Epoch: 11/200,  Loss: 0.0184
........................................Epoch: 11/200,  Loss: 0.0257
........................................Epoch: 11/200,  Loss: 0.0199
........................................Epoch: 11/200,  Loss: 0.0271
........................................Epoch: 11/200,  Loss: 0.0242
.............................           Epoch 11 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 62.6680 %
Confusion matrix
[[229   0   1   7   6  10   2  34   0   2]
 [  0 258   6   6   3   0   0   5   1  11]
 [ 13  19 146  38  12   3   3  13   1  43]
 [ 38   8  17 147   8   3  30  31   1   7]
 [  2   0   0   0 261  15   0   8   4   0]
 [ 25   0   1   9  34  79  13 113  16   0]
 [ 94   0   1  14   0   0 139  42   0   0]
 [ 40   0   1   8   9  10  13 205   3   0]
 [  1   4   0   9  26  22   1  34 193   0]
 [  3  18  48  31  13   2   1  10   3 161]]
........................................Epoch: 12/200,  Loss: 0.0223
........................................Epoch: 12/200,  Loss: 0.0274
........................................Epoch: 12/200,  Loss: 0.0306
........................................Epoch: 12/200,  Loss: 0.0252
........................................Epoch: 12/200,  Loss: 0.0232
........................................Epoch: 12/200,  Loss: 0.0226
.............................           Epoch 12 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 62.9783 %
Confusion matrix
[[216   0   1   3   2  41   2  20   1   5]
 [  0 259   6   3   2   1   0   1   2  16]
 [  6  20 135   8   3  15   2   3   3  96]
 [ 45   8  30  92   5  14  31  17   9  39]
 [  1   0   1   0 226  49   0   4   4   5]
 [ 19   1   2   4  13 184  12  33  21   1]
 [ 92   2   0   5   0  15 143  33   0   0]
 [ 36   0   2   3   3  94   8 138   3   2]
 [  0   3   1   2  20  44   0   9 210   1]
 [  4  14  25  11   4   4   1   1   2 224]]
........................................Epoch: 13/200,  Loss: 0.0322
........................................Epoch: 13/200,  Loss: 0.0234
........................................Epoch: 13/200,  Loss: 0.0227
........................................Epoch: 13/200,  Loss: 0.0269
........................................Epoch: 13/200,  Loss: 0.0227
........................................Epoch: 13/200,  Loss: 0.0218
.............................           Epoch 13 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 64.9087 %
Confusion matrix
[[202   0   2   7   3   7  55  10   1   4]
 [  0 244  20   7   1   0   0   1   3  14]
 [  8   8 191  35   4   2   6   2   0  35]
 [ 29   6  26 148   4   2  54   7   5   9]
 [  4   0   2   1 254  13   0   7   7   2]
 [ 36   0   1   8  31  98  25  64  26   1]
 [ 35   1   2  11   0   0 226  15   0   0]
 [ 55   0   2  17   6  20  46 139   4   0]
 [  3   4   0   8  26  18   2  10 219   0]
 [  3  12  77  22   7   1   3   0   3 162]]
........................................Epoch: 14/200,  Loss: 0.0246
........................................Epoch: 14/200,  Loss: 0.0230
........................................Epoch: 14/200,  Loss: 0.0275
........................................Epoch: 14/200,  Loss: 0.0275
........................................Epoch: 14/200,  Loss: 0.0262
........................................Epoch: 14/200,  Loss: 0.0251
.............................           Epoch 14 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 64.8742 %
Confusion matrix
[[206   0   4  11   1   8  48  10   2   1]
 [  0 258  14  11   0   0   0   0   2   5]
 [  8  18 208  37   2   2   4   2   0  10]
 [ 29   7  29 163   3   1  41   8   4   5]
 [  8   0   3  12 200  39   0   9  18   1]
 [ 35   1   3  10  10 130  22  39  39   1]
 [ 49   1   1  13   0   1 208  16   1   0]
 [ 62   1   2  18   1  20  31 146   8   0]
 [  5   4   1   7  11  18   2   4 238   0]
 [  2  16 106  31   2   2   1   3   2 125]]
........................................Epoch: 15/200,  Loss: 0.0213
........................................Epoch: 15/200,  Loss: 0.0227
........................................Epoch: 15/200,  Loss: 0.0166
........................................Epoch: 15/200,  Loss: 0.0227
........................................Epoch: 15/200,  Loss: 0.0237
........................................Epoch: 15/200,  Loss: 0.0230
.............................           Epoch 15 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 65.2189 %
Confusion matrix
[[192   0   1   3   3  30  31  24   1   6]
 [  0 256   4   8   0   1   0   2   2  17]
 [  4  19 131  32   2  10   2   4   1  86]
 [ 19   6  13 139   3  13  29  22   7  39]
 [  0   1   0   3 230  39   0   5   5   7]
 [ 16   0   1   1  15 183  15  35  21   3]
 [ 45   1   0  11   0  14 174  43   2   0]
 [ 24   0   2   9   6  93   9 140   4   2]
 [  0   4   0   4  16  42   0   8 213   3]
 [  2  16  15  14   4   1   0   3   1 234]]
........................................Epoch: 16/200,  Loss: 0.0175
........................................Epoch: 16/200,  Loss: 0.0304
........................................Epoch: 16/200,  Loss: 0.0241
........................................Epoch: 16/200,  Loss: 0.0238
........................................Epoch: 16/200,  Loss: 0.0189
........................................Epoch: 16/200,  Loss: 0.0278
.............................           Epoch 16 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 65.6325 %
Confusion matrix
[[161   0   6   6   3  18  45  46   1   5]
 [  0 274   4   0   0   1   0   1   1   9]
 [  4  32 185   9   0   3   2   5   3  48]
 [ 17  12  43 119   5  10  33  21  10  20]
 [  1   0   0   2 206  51   0  12  11   7]
 [ 15   0   3   4  10 159  17  60  21   1]
 [ 32   0   0  16   0   9 197  34   2   0]
 [ 19   1   5   9   3  46  22 178   4   2]
 [  0   5   2   4   7  34   0   6 232   0]
 [  1  34  42   8   3   2   1   4   2 193]]
........................................Epoch: 17/200,  Loss: 0.0202
........................................Epoch: 17/200,  Loss: 0.0209
........................................Epoch: 17/200,  Loss: 0.0217
........................................Epoch: 17/200,  Loss: 0.0223
........................................Epoch: 17/200,  Loss: 0.0252
........................................Epoch: 17/200,  Loss: 0.0146
.............................           Epoch 17 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 65.5981 %
Confusion matrix
[[231   0   5  11   4  14  10  11   3   2]
 [  1 270   6   5   1   0   0   0   2   5]
 [  6  27 185  31   3   3   4   2   1  29]
 [ 32  10  26 163   5   2  26  10   5  11]
 [  3   0   0   5 243  18   0   3  16   2]
 [ 36   1   1   9  15 133  17  27  49   2]
 [ 77   1   0  21   0   4 168  18   1   0]
 [ 65   3   3  19   9  42  26 114   8   0]
 [  2   5   0   5  22  12   0   4 240   0]
 [  3  41  58  19   7   1   1   2   2 156]]
........................................Epoch: 18/200,  Loss: 0.0217
........................................Epoch: 18/200,  Loss: 0.0194
........................................Epoch: 18/200,  Loss: 0.0282
........................................Epoch: 18/200,  Loss: 0.0219
........................................Epoch: 18/200,  Loss: 0.0189
........................................Epoch: 18/200,  Loss: 0.0110
.............................           Epoch 18 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 65.4947 %
Confusion matrix
[[160   0   2  20   3  11  68  22   0   5]
 [  0 263   8   4   1   0   0   1   2  11]
 [  3  21 177  33   1   0   4   1   0  51]
 [ 11   7  31 161   5   2  44  12   0  17]
 [  1   0   2   7 242  18   0   9   4   7]
 [ 21   1   2  14  22 118  21  74  14   3]
 [ 30   1   2  16   0   1 215  25   0   0]
 [ 28   1   4  23   4  19  29 177   3   1]
 [  1   5   0  19  27  36   2  13 186   1]
 [  1  21  35  20   6   1   1   3   1 201]]
........................................Epoch: 19/200,  Loss: 0.0203
........................................Epoch: 19/200,  Loss: 0.0196
........................................Epoch: 19/200,  Loss: 0.0263
........................................Epoch: 19/200,  Loss: 0.0228
........................................Epoch: 19/200,  Loss: 0.0263
........................................Epoch: 19/200,  Loss: 0.0213
.............................           Epoch 19 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 65.1499 %
Confusion matrix
[[232   0   0   6   1  11  24  12   1   4]
 [  1 247   3  14   1   0   3   3   3  15]
 [ 15  14 146  28   2   2   9   5   0  70]
 [ 37   6  14 133   4   3  51  13   5  24]
 [  6   0   0   3 233  27   0   7   9   5]
 [ 46   0   1   6  10 150  19  28  29   1]
 [ 65   0   0   6   0  11 195  13   0   0]
 [ 58   0   1  12   1  57  33 124   3   0]
 [  2   4   0   5  15  31   5  13 215   0]
 [  6  13  20  21   8   1   1   3   2 215]]
........................................Epoch: 20/200,  Loss: 0.0262
........................................Epoch: 20/200,  Loss: 0.0167
........................................Epoch: 20/200,  Loss: 0.0184
........................................Epoch: 20/200,  Loss: 0.0234
........................................Epoch: 20/200,  Loss: 0.0230
........................................Epoch: 20/200,  Loss: 0.0182
.............................           Epoch 20 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 66.4254 %
Confusion matrix
[[209   0   2   6   1  27  28  14   1   3]
 [  0 231  29  10   2   1   2   1   3  11]
 [  8  12 203  18   9   7   6   5   0  23]
 [ 31   6  34 138   7   4  45  12   7   6]
 [  3   0   0   1 249  21   0   5  10   1]
 [ 21   1   2   2  20 174  18  24  28   0]
 [ 52   0   0   8   0  15 197  18   0   0]
 [ 34   1   2  14   2  82  24 126   4   0]
 [  2   3   0   4  25  18   1   9 228   0]
 [  3  13  62  24   9   1   1   3   2 172]]
........................................Epoch: 21/200,  Loss: 0.0204
........................................Epoch: 21/200,  Loss: 0.0186
........................................Epoch: 21/200,  Loss: 0.0242
........................................Epoch: 21/200,  Loss: 0.0256
........................................Epoch: 21/200,  Loss: 0.0176
........................................Epoch: 21/200,  Loss: 0.0214
.............................           Epoch 21 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 65.4257 %
Confusion matrix
[[157   1   7  11   5  28  17  61   1   3]
 [  0 278   6   0   0   0   0   0   2   4]
 [  3  29 216   6   3   5   2   3   2  22]
 [ 19  11  45 144   6   4  21  25   7   8]
 [  0   1   1   0 257  17   0   4   8   2]
 [ 17   3   1   2  41 157   8  39  22   0]
 [ 41   1   1  17   0  15 153  61   1   0]
 [ 14   3   3  11  15  61   8 171   3   0]
 [  0   8   0   4  23  18   0  10 226   1]
 [  2  40  85  10   9   1   0   3   1 139]]
........................................Epoch: 22/200,  Loss: 0.0244
........................................Epoch: 22/200,  Loss: 0.0179
........................................Epoch: 22/200,  Loss: 0.0147
........................................Epoch: 22/200,  Loss: 0.0280
........................................Epoch: 22/200,  Loss: 0.0299
........................................Epoch: 22/200,  Loss: 0.0251
.............................           Epoch 22 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 65.2534 %
Confusion matrix
[[228   0   1   3   3  22  10  20   1   3]
 [  2 250   7   7   1   0   0   3   2  18]
 [  9  10 158  28   4   8   0  11   0  63]
 [ 36   5  19 148   4  12  20  24   5  17]
 [  4   0   0   1 232  40   0   8   4   1]
 [ 28   1   1   2  13 183  11  40  11   0]
 [ 76   2   0  14   0  12 141  45   0   0]
 [ 44   1   1  10   3  63   9 156   2   0]
 [  3   2   0   5  14  53   1  18 193   1]
 [  4  11  35  21   7   2   0   4   2 204]]
........................................Epoch: 23/200,  Loss: 0.0170
........................................Epoch: 23/200,  Loss: 0.0209
........................................Epoch: 23/200,  Loss: 0.0328
........................................Epoch: 23/200,  Loss: 0.0150
........................................Epoch: 23/200,  Loss: 0.0149
........................................Epoch: 23/200,  Loss: 0.0222
.............................           Epoch 23 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 65.9083 %
Confusion matrix
[[152   1   4  14   7  25  19  67   1   1]
 [  0 249  15   7   2   0   0   1   5  11]
 [  2  14 197  22  13   7   0   4   3  29]
 [ 15   5  30 155   9  11  19  25   7  14]
 [  0   0   0   1 259  18   0   3   9   0]
 [  9   1   1   4  39 152   9  45  30   0]
 [ 32   1   1  19   1  11 159  65   1   0]
 [ 15   0   1  11  13  64   7 169   9   0]
 [  0   4   0   4  21  18   0   4 239   0]
 [  1  12  58  17  13   1   1   3   3 181]]
........................................Epoch: 24/200,  Loss: 0.0268
........................................Epoch: 24/200,  Loss: 0.0183
........................................Epoch: 24/200,  Loss: 0.0174
........................................Epoch: 24/200,  Loss: 0.0259
........................................Epoch: 24/200,  Loss: 0.0342
........................................Epoch: 24/200,  Loss: 0.0272
.............................           Epoch 24 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 66.6667 %
Confusion matrix
[[201   0   3   6   1  17   6  54   1   2]
 [  0 247  16   6   0   0   0   3   5  13]
 [  5   9 216  13   0   1   1   9   3  34]
 [ 27   4  31 148   3   5  20  31   5  16]
 [  2   0   4   3 218  34   0  13  13   3]
 [ 17   1   1   3   5 132  10  88  32   1]
 [ 70   1   1  12   0   7 149  50   0   0]
 [ 26   0   1   7   2  37   8 204   4   0]
 [  1   3   0   3  10  20   0  22 231   0]
 [  1  12  58  19   3   1   0   6   2 188]]
........................................Epoch: 25/200,  Loss: 0.0268
........................................Epoch: 25/200,  Loss: 0.0286
........................................Epoch: 25/200,  Loss: 0.0243
........................................Epoch: 25/200,  Loss: 0.0170
........................................Epoch: 25/200,  Loss: 0.0213
........................................Epoch: 25/200,  Loss: 0.0236
.............................           Epoch 25 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 66.3909 %
Confusion matrix
[[219   0   3  11   3   9  28  15   1   2]
 [  0 273   6   6   0   0   0   0   0   5]
 [  5  27 203  26   1   1   4   3   1  20]
 [ 22   8  30 162   3   2  35  13   4  11]
 [  3   1   1   6 235  23   0   7   7   7]
 [ 39   2   2   7  20 131  22  49  17   1]
 [ 58   2   0  10   0   9 191  20   0   0]
 [ 53   3   1  13   3  33  23 156   4   0]
 [  4   9   1   3  23  24   0  11 213   2]
 [  3  30  89  19   3   1   0   1   1 143]]
........................................Epoch: 26/200,  Loss: 0.0205
........................................Epoch: 26/200,  Loss: 0.0226
........................................Epoch: 26/200,  Loss: 0.0317
........................................Epoch: 26/200,  Loss: 0.0252
........................................Epoch: 26/200,  Loss: 0.0289
........................................Epoch: 26/200,  Loss: 0.0199
.............................           Epoch 26 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 66.1151 %
Confusion matrix
[[225   0   2   7   3  22   9  20   0   3]
 [  0 258  10   4   0   1   0   1   2  14]
 [  6  11 206  13   1   4   4   2   0  44]
 [ 33   5  34 158   4  12  19  15   0  10]
 [  2   0   1   0 229  46   0   9   0   3]
 [ 34   1   2   7  11 178  13  37   6   1]
 [ 85   0   1  15   0  17 148  24   0   0]
 [ 50   0   3  11   4  55   7 156   3   0]
 [  2   4   1  11  13  68   1  19 169   2]
 [  3   9  57  18   6   2   0   3   1 191]]
........................................Epoch: 27/200,  Loss: 0.0233
........................................Epoch: 27/200,  Loss: 0.0216
........................................Epoch: 27/200,  Loss: 0.0210
........................................Epoch: 27/200,  Loss: 0.0179
........................................Epoch: 27/200,  Loss: 0.0297
........................................Epoch: 27/200,  Loss: 0.0268
.............................           Epoch 27 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 65.2189 %
Confusion matrix
[[181   0   1   4   6   8  76  11   1   3]
 [  0 251   8  10   3   0   1   3   6   8]
 [ 14  11 185  35  10   1   8   4   3  20]
 [ 28   4  25 150   6   2  52  15   5   3]
 [  5   0   0   0 260   9   0   7   9   0]
 [ 41   1   1   4  53  95  27  47  21   0]
 [ 27   1   0   8   0   3 237  14   0   0]
 [ 47   0   1   6  11  23  46 151   4   0]
 [  2   3   0   7  23  11   2  10 232   0]
 [  5  15  66  29  14   2   2   4   3 150]]
........................................Epoch: 28/200,  Loss: 0.0291
........................................Epoch: 28/200,  Loss: 0.0207
........................................Epoch: 28/200,  Loss: 0.0265
........................................Epoch: 28/200,  Loss: 0.0193
........................................Epoch: 28/200,  Loss: 0.0293
........................................Epoch: 28/200,  Loss: 0.0208
.............................           Epoch 28 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 67.7008 %
Confusion matrix
[[198   0   0  10   0  20  15  44   1   3]
 [  0 250   8   5   0   0   0   3   3  21]
 [  6   9 169  19   0   4   3   9   1  71]
 [ 26   4  18 151   2   4  28  26   4  27]
 [  0   0   1   4 198  52   0  18  11   6]
 [ 23   0   1   5   5 166   9  55  25   1]
 [ 53   0   0  11   0  16 160  50   0   0]
 [ 23   0   1   8   1  42   7 203   4   0]
 [  2   4   0   7   5  22   0  15 232   3]
 [  3   7  20  15   2   1   0   4   1 237]]
........................................Epoch: 29/200,  Loss: 0.0167
........................................Epoch: 29/200,  Loss: 0.0177
........................................Epoch: 29/200,  Loss: 0.0132
........................................Epoch: 29/200,  Loss: 0.0211
........................................Epoch: 29/200,  Loss: 0.0281
........................................Epoch: 29/200,  Loss: 0.0184
.............................           Epoch 29 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.5626 %
Confusion matrix
[[204   0   6   5   2  23  25  21   1   4]
 [  0 253   5   3   0   0   0   1   6  22]
 [  4   9 181  16   3   4   1   2   5  66]
 [ 25   4  33 151   6   8  22  10   6  25]
 [  0   0   0   2 244  19   0   7  14   4]
 [ 27   1   1  10  13 163  13  28  32   2]
 [ 54   1   1  26   0   4 174  29   1   0]
 [ 31   0   2   9   8  56  16 158   8   1]
 [  1   3   0   6  12  19   0   3 245   1]
 [  2  11  35  11   7   2   1   2   3 216]]
........................................Epoch: 30/200,  Loss: 0.0305
........................................Epoch: 30/200,  Loss: 0.0205
........................................Epoch: 30/200,  Loss: 0.0205
........................................Epoch: 30/200,  Loss: 0.0238
........................................Epoch: 30/200,  Loss: 0.0213
........................................Epoch: 30/200,  Loss: 0.0254
.............................           Epoch 30 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 65.7360 %
Confusion matrix
[[207   0   0   0   1  22  22  34   1   4]
 [  0 256  11   3   2   0   1   2   3  12]
 [  9   9 194  10   2  10   4  11   2  40]
 [ 33   4  27 112   4  20  38  34   4  14]
 [  1   0   0   1 205  66   0  10   4   3]
 [ 21   1   1   1   7 197  10  42  10   0]
 [ 49   1   0   5   0  14 175  46   0   0]
 [ 33   0   1   2   0  66  11 173   3   0]
 [  2   5   1   2   9  55   0  21 195   0]
 [  5   8  49  10   4  11   0   8   2 193]]
........................................Epoch: 31/200,  Loss: 0.0264
........................................Epoch: 31/200,  Loss: 0.0180
........................................Epoch: 31/200,  Loss: 0.0217
........................................Epoch: 31/200,  Loss: 0.0242
........................................Epoch: 31/200,  Loss: 0.0318
........................................Epoch: 31/200,  Loss: 0.0166
.............................           Epoch 31 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 67.1148 %
Confusion matrix
[[236   0   0   3   5  12   9  22   1   3]
 [  1 262   6   3   3   0   0   3   2  10]
 [  6  14 172  29  10   2   5   9   3  41]
 [ 36   6  15 154   9   3  30  23   4  10]
 [  2   0   0   2 249  18   0  11   5   3]
 [ 36   1   1   3  23 152  11  53  10   0]
 [ 81   0   0   8   0   8 162  31   0   0]
 [ 50   1   1   7   5  33   7 182   3   0]
 [  2   6   0   6  28  37   2  14 194   1]
 [  3  22  40  21  11   2   0   5   2 184]]
........................................Epoch: 32/200,  Loss: 0.0253
........................................Epoch: 32/200,  Loss: 0.0164
........................................Epoch: 32/200,  Loss: 0.0254
........................................Epoch: 32/200,  Loss: 0.0239
........................................Epoch: 32/200,  Loss: 0.0213
........................................Epoch: 32/200,  Loss: 0.0198
.............................           Epoch 32 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.8383 %
Confusion matrix
[[203   0   4   4   4  20  23  30   1   2]
 [  0 260  10   0   1   0   0   1   3  15]
 [  5  16 204  16   1   3   1   2   3  40]
 [ 23   7  43 140   4   7  26  15   8  17]
 [  0   0   3   1 249  15   0   7  11   4]
 [ 20   1   2   6  19 154  11  41  34   2]
 [ 45   0   0  17   0   7 176  44   1   0]
 [ 29   0   3  13   6  48  11 171   8   0]
 [  1   5   0   6  18  14   0   6 239   1]
 [  3  12  46  15   8   1   0   2   2 201]]
........................................Epoch: 33/200,  Loss: 0.0131
........................................Epoch: 33/200,  Loss: 0.0244
........................................Epoch: 33/200,  Loss: 0.0224
........................................Epoch: 33/200,  Loss: 0.0212
........................................Epoch: 33/200,  Loss: 0.0244
........................................Epoch: 33/200,  Loss: 0.0139
.............................           Epoch 33 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.2179 %
Confusion matrix
[[237   0   2   5   6  13  10  15   0   3]
 [  1 259   3   3   4   0   0   0   2  18]
 [  5  18 166  16   9   5   5   1   1  65]
 [ 35   4  27 154   6   3  20  14   4  23]
 [  1   0   0   2 269   6   0   6   4   2]
 [ 40   1   1   8  40 146   6  34  12   2]
 [ 77   0   1  20   0   7 156  28   1   0]
 [ 50   0   2  13  17  35   5 162   4   1]
 [  3   5   0  12  28  28   0   8 205   1]
 [  2   6  31  13   9   1   0   2   1 225]]
........................................Epoch: 34/200,  Loss: 0.0150
........................................Epoch: 34/200,  Loss: 0.0356
........................................Epoch: 34/200,  Loss: 0.0204
........................................Epoch: 34/200,  Loss: 0.0207
........................................Epoch: 34/200,  Loss: 0.0151
........................................Epoch: 34/200,  Loss: 0.0111
.............................           Epoch 34 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.2179 %
Confusion matrix
[[204   0   1   2   4  24  19  34   2   1]
 [  0 252  13   1   2   0   1   3   6  12]
 [  5  10 191  22   7  11   6   6   9  24]
 [ 30   6  19 156   7   8  33  18   8   5]
 [  0   0   0   1 241  25   0  10  12   1]
 [ 19   1   1   5  12 187  13  34  18   0]
 [ 51   0   0  16   0   9 168  46   0   0]
 [ 21   0   2   7   6  66  11 171   5   0]
 [  1   3   0   5  11  26   0   8 236   0]
 [  3  16  51  29   9   3   0   3   3 173]]
........................................Epoch: 35/200,  Loss: 0.0237
........................................Epoch: 35/200,  Loss: 0.0214
........................................Epoch: 35/200,  Loss: 0.0235
........................................Epoch: 35/200,  Loss: 0.0185
........................................Epoch: 35/200,  Loss: 0.0182
........................................Epoch: 35/200,  Loss: 0.0186
.............................           Epoch 35 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 69.0107 %
Confusion matrix
[[205   0   1   5   6  20  24  25   2   3]
 [  0 263   5   1   0   0   3   1   5  12]
 [  6  19 180  18   4   7   5   5   3  44]
 [ 28   9  26 146   5   6  33  15   6  16]
 [  0   0   0   1 255  16   0   8   9   1]
 [ 20   1   1   4  26 158  17  38  22   3]
 [ 45   1   0  14   1   7 193  29   0   0]
 [ 30   0   2   8  10  53  16 164   6   0]
 [  1   5   0   6  18  17   0   4 238   1]
 [  3  20  36  16   8   2   0   2   3 200]]
........................................Epoch: 36/200,  Loss: 0.0173
........................................Epoch: 36/200,  Loss: 0.0190
........................................Epoch: 36/200,  Loss: 0.0115
........................................Epoch: 36/200,  Loss: 0.0176
........................................Epoch: 36/200,  Loss: 0.0339
........................................Epoch: 36/200,  Loss: 0.0222
.............................           Epoch 36 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 69.2520 %
Confusion matrix
[[211   0   1  10   3  18   7  36   2   3]
 [  0 268   3   7   0   0   0   0   1  11]
 [  3  17 203  13   0   1   3   4   3  44]
 [ 25  10  36 157   3   7  15  16   7  14]
 [  0   0   2   2 224  34   0  10  14   4]
 [ 28   1   2   6  12 166   8  47  20   0]
 [ 70   1   2  10   0   6 155  45   1   0]
 [ 30   0   3  14   2  36   5 194   5   0]
 [  2   5   1   3   7  20   0   9 242   1]
 [  3  19  49  17   5   2   0   4   2 189]]
........................................Epoch: 37/200,  Loss: 0.0230
........................................Epoch: 37/200,  Loss: 0.0221
........................................Epoch: 37/200,  Loss: 0.0243
........................................Epoch: 37/200,  Loss: 0.0216
........................................Epoch: 37/200,  Loss: 0.0177
........................................Epoch: 37/200,  Loss: 0.0327
.............................           Epoch 37 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 67.8042 %
Confusion matrix
[[235   0   0   1   5  14  14  19   1   2]
 [  1 246  18   5   5   1   1   1   3   9]
 [ 11  10 179  27   4   7   3   7   6  37]
 [ 41   3  17 159   8   4  25  16   8   9]
 [  0   0   0   1 244  19   0  12  13   1]
 [ 29   1   1   5  24 132  15  55  27   1]
 [ 73   0   0  12   0   3 168  33   1   0]
 [ 50   0   2   6   7  31   6 183   4   0]
 [  2   4   0   6  13  15   0   8 242   0]
 [  6  16  52  17   8   2   0   7   3 179]]
........................................Epoch: 38/200,  Loss: 0.0174
........................................Epoch: 38/200,  Loss: 0.0216
........................................Epoch: 38/200,  Loss: 0.0179
........................................Epoch: 38/200,  Loss: 0.0156
........................................Epoch: 38/200,  Loss: 0.0149
........................................Epoch: 38/200,  Loss: 0.0200
.............................           Epoch 38 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.9762 %
Confusion matrix
[[199   0   2   2   2  32  26  25   1   2]
 [  0 273   3   1   1   1   0   1   3   7]
 [  5  21 176  18   0   7   3   9   2  50]
 [ 21   9  18 144   3   9  42  22   4  18]
 [  1   0   2   2 237  28   0  11   7   2]
 [ 20   1   1   3  14 195  14  24  17   1]
 [ 43   1   0   2   0  15 194  34   1   0]
 [ 23   0   3   7   4  72  14 161   5   0]
 [  1   4   0   5  14  30   1   4 230   1]
 [  4  30  33  19   5   1   0   4   2 192]]
........................................Epoch: 39/200,  Loss: 0.0225
........................................Epoch: 39/200,  Loss: 0.0157
........................................Epoch: 39/200,  Loss: 0.0189
........................................Epoch: 39/200,  Loss: 0.0162
........................................Epoch: 39/200,  Loss: 0.0182
........................................Epoch: 39/200,  Loss: 0.0222
.............................           Epoch 39 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 67.8387 %
Confusion matrix
[[228   0   0   4   2  22  17  14   1   3]
 [  1 260   2   3   4   1   0   1   3  15]
 [  6  16 121  10   7   6   3  12   4 106]
 [ 42   3  11 129   5   5  33  16   7  39]
 [  0   0   0   2 246  25   0   8   6   3]
 [ 24   1   0   2  17 186  13  29  15   3]
 [ 70   1   0   8   0  14 180  16   1   0]
 [ 42   1   1   7   7  57  14 156   3   1]
 [  1   4   0   3  23  28   1   8 220   2]
 [  3   9   8  12   9   1   0   4   2 242]]
........................................Epoch: 40/200,  Loss: 0.0127
........................................Epoch: 40/200,  Loss: 0.0193
........................................Epoch: 40/200,  Loss: 0.0221
........................................Epoch: 40/200,  Loss: 0.0206
........................................Epoch: 40/200,  Loss: 0.0306
........................................Epoch: 40/200,  Loss: 0.0309
.............................           Epoch 40 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 67.2871 %
Confusion matrix
[[186   0   3   2   2  35  13  48   1   1]
 [  0 249  16   0   3   1   0   2   6  13]
 [  5  10 208  14   3   5   2   8   5  31]
 [ 33   6  26 128   6  11  26  29   8  17]
 [  0   0   2   1 222  31   0  11  21   2]
 [ 13   1   1   5   9 184  10  38  29   0]
 [ 50   1   0  10   0  13 161  55   0   0]
 [ 14   0   1   6   2  65   6 190   5   0]
 [  0   4   0   5  13  27   0   8 232   1]
 [  2  15  47  13   9   2   0   7   3 192]]
........................................Epoch: 41/200,  Loss: 0.0157
........................................Epoch: 41/200,  Loss: 0.0185
........................................Epoch: 41/200,  Loss: 0.0177
........................................Epoch: 41/200,  Loss: 0.0345
........................................Epoch: 41/200,  Loss: 0.0291
........................................Epoch: 41/200,  Loss: 0.0223
.............................           Epoch 41 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 69.7001 %
Confusion matrix
[[200   0   1   7   3  20  45  13   1   1]
 [  1 256   8   8   1   0   0   3   2  11]
 [  5   7 205  31   1   5   4   0   3  30]
 [ 18   3  23 165   4   9  47  10   2   9]
 [  4   0   0   2 249  24   0   3   5   3]
 [ 30   1   2   3  14 189  18  22  11   0]
 [ 29   1   0   4   0  12 231  13   0   0]
 [ 44   0   4  11   6  54  29 138   3   0]
 [  2   5   1  12  18  38   2   6 205   1]
 [  4  17  52  22   6   1   0   2   2 184]]
........................................Epoch: 42/200,  Loss: 0.0240
........................................Epoch: 42/200,  Loss: 0.0276
........................................Epoch: 42/200,  Loss: 0.0168
........................................Epoch: 42/200,  Loss: 0.0308
........................................Epoch: 42/200,  Loss: 0.0195
........................................Epoch: 42/200,  Loss: 0.0194
.............................           Epoch 42 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.0800 %
Confusion matrix
[[227   1   0   2   2  25  10  19   1   4]
 [  0 280   3   0   0   0   0   0   3   4]
 [  7  37 188   5   2   4   0   3   2  43]
 [ 33  13  29 136   5   6  21  20   8  19]
 [  2   0   1   0 238  24   0   5  15   5]
 [ 30   1   1   3  16 174   5  31  29   0]
 [ 78   1   2   8   0  31 135  33   2   0]
 [ 31   2   4   7   5  54   5 175   4   2]
 [  1   6   0   5  12  19   0   7 238   2]
 [  3  57  24  11   5   1   0   3   2 184]]
........................................Epoch: 43/200,  Loss: 0.0258
........................................Epoch: 43/200,  Loss: 0.0177
........................................Epoch: 43/200,  Loss: 0.0172
........................................Epoch: 43/200,  Loss: 0.0163
........................................Epoch: 43/200,  Loss: 0.0206
........................................Epoch: 43/200,  Loss: 0.0249
.............................           Epoch 43 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 67.2871 %
Confusion matrix
[[199   0   2   2   1  18  18  49   0   2]
 [  1 259  11   3   1   0   2   2   2   9]
 [  8  10 217  16   0   6   2   7   1  24]
 [ 28   4  34 144   5  11  27  29   1   7]
 [  2   0   3   1 220  38   0  15   7   4]
 [ 18   1   2   2  12 162  11  71  11   0]
 [ 45   0   0  12   0  14 175  44   0   0]
 [ 18   0   2   7   1  34  10 213   4   0]
 [  1   4   0   6  12  38   1  21 206   1]
 [  2  19  75  22   4   2   0   7   2 157]]
........................................Epoch: 44/200,  Loss: 0.0152
........................................Epoch: 44/200,  Loss: 0.0171
........................................Epoch: 44/200,  Loss: 0.0201
........................................Epoch: 44/200,  Loss: 0.0180
........................................Epoch: 44/200,  Loss: 0.0227
........................................Epoch: 44/200,  Loss: 0.0309
.............................           Epoch 44 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 69.8725 %
Confusion matrix
[[202   0   5  14   4  13  22  28   1   2]
 [  0 265  12   1   0   0   0   0   3   9]
 [  3  17 214  16   1   2   2   3   1  32]
 [ 15   6  33 178   4   0  31   7   3  13]
 [  1   0   3   7 245  15   0   5  11   3]
 [ 31   1   5   8  32 145  19  28  18   3]
 [ 44   1   0  14   0   2 208  20   1   0]
 [ 29   0   5  18   9  36  17 169   4   2]
 [  1   6   2  10  20  21   1   6 222   1]
 [  2  31  48  19   7   1   0   1   2 179]]
........................................Epoch: 45/200,  Loss: 0.0217
........................................Epoch: 45/200,  Loss: 0.0264
........................................Epoch: 45/200,  Loss: 0.0151
........................................Epoch: 45/200,  Loss: 0.0261
........................................Epoch: 45/200,  Loss: 0.0202
........................................Epoch: 45/200,  Loss: 0.0244
.............................           Epoch 45 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.0455 %
Confusion matrix
[[158   1   1   4   4  32  15  73   2   1]
 [  0 269   4   1   2   1   0   2   1  10]
 [  2  21 175  17   7  15   2  11   0  41]
 [ 13   7  15 150   5  18  26  36   3  17]
 [  0   0   0   0 244  25   0  10   8   3]
 [ 10   1   1   2  21 194  12  40   8   1]
 [ 37   1   0  12   0  16 175  49   0   0]
 [ 13   0   1   4   5  52  11 200   3   0]
 [  0   5   0   6  20  33   1  13 210   2]
 [  1  30  29  11   9   3   0   8   0 199]]
........................................Epoch: 46/200,  Loss: 0.0180
........................................Epoch: 46/200,  Loss: 0.0212
........................................Epoch: 46/200,  Loss: 0.0266
........................................Epoch: 46/200,  Loss: 0.0187
........................................Epoch: 46/200,  Loss: 0.0137
........................................Epoch: 46/200,  Loss: 0.0232
.............................           Epoch 46 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.5970 %
Confusion matrix
[[231   0   1  11   4  10  14  18   1   1]
 [  2 256  10   7   2   0   2   0   4   7]
 [ 10  14 217  20   3   0   0   6   2  19]
 [ 41   5  23 171   5   2  17  12   8   6]
 [  1   0   2   3 234  23   0   9  16   2]
 [ 32   0   1   6  19 153   8  44  27   0]
 [ 67   1   2  12   0  15 174  17   2   0]
 [ 68   0   2  14   4  35   9 152   5   0]
 [  1   3   2   6  17  13   0   5 242   1]
 [  5  17  71  23   7   1   0   4   2 160]]
........................................Epoch: 47/200,  Loss: 0.0203
........................................Epoch: 47/200,  Loss: 0.0220
........................................Epoch: 47/200,  Loss: 0.0155
........................................Epoch: 47/200,  Loss: 0.0188
........................................Epoch: 47/200,  Loss: 0.0140
........................................Epoch: 47/200,  Loss: 0.0190
.............................           Epoch 47 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.8039 %
Confusion matrix
[[204   0   0  12   4  21  13  32   1   4]
 [  1 254   2   6   2   0   2   2   2  19]
 [  5  11 160  19   3   6   2   2   1  82]
 [ 19   4  15 167   6  12  18  15   1  33]
 [  0   0   0   0 252  23   0   7   3   5]
 [ 25   1   1   4  19 199   4  23  11   3]
 [ 54   0   0  14   0  25 164  32   1   0]
 [ 32   0   2  12   6  67   7 158   3   2]
 [  2   2   0   5  21  39   0   4 211   6]
 [  3  11  22  15   7   2   0   2   1 227]]
........................................Epoch: 48/200,  Loss: 0.0232
........................................Epoch: 48/200,  Loss: 0.0208
........................................Epoch: 48/200,  Loss: 0.0191
........................................Epoch: 48/200,  Loss: 0.0249
........................................Epoch: 48/200,  Loss: 0.0193
........................................Epoch: 48/200,  Loss: 0.0225
.............................           Epoch 48 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 67.4940 %
Confusion matrix
[[210   0   3   4   5  14  45  10   0   0]
 [  0 255  15   5   2   0   1   1   4   7]
 [  7  14 185  34   9   7   4   4   3  24]
 [ 25   6  26 168   7   8  33   6   3   8]
 [  1   0   0   1 258  17   0   8   4   1]
 [ 28   1   1   3  28 178  17  22  12   0]
 [ 44   0   0  11   1  10 217   7   0   0]
 [ 46   0   2  14   8  64  22 128   5   0]
 [  1   5   0   7  30  34   0   5 206   2]
 [  2  25  67  23  11   3   1   4   1 153]]
........................................Epoch: 49/200,  Loss: 0.0142
........................................Epoch: 49/200,  Loss: 0.0206
........................................Epoch: 49/200,  Loss: 0.0201
........................................Epoch: 49/200,  Loss: 0.0286
........................................Epoch: 49/200,  Loss: 0.0188
........................................Epoch: 49/200,  Loss: 0.0298
.............................           Epoch 49 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.0800 %
Confusion matrix
[[213   0   1   3   5   9  41  17   1   1]
 [  0 267   2   8   2   0   1   2   3   5]
 [  7  20 147  45   6  12   5   9   5  35]
 [ 30   8   9 154   5  11  43  14   7   9]
 [  3   0   0   1 237  35   0   7   6   1]
 [ 31   1   0   3  11 179  14  35  16   0]
 [ 44   0   0   7   0  12 207  18   2   0]
 [ 38   0   1   5   6  39  20 176   4   0]
 [  2   4   0   4  14  34   0   5 227   0]
 [  4  34  35  30   8   2   1   6   2 168]]
........................................Epoch: 50/200,  Loss: 0.0188
........................................Epoch: 50/200,  Loss: 0.0229
........................................Epoch: 50/200,  Loss: 0.0161
........................................Epoch: 50/200,  Loss: 0.0203
........................................Epoch: 50/200,  Loss: 0.0232
........................................Epoch: 50/200,  Loss: 0.0220
.............................           Epoch 50 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 70.5274 %
Confusion matrix
[[225   1   2  11   4  17  19  10   0   2]
 [  0 268   4   5   1   0   0   1   1  10]
 [  6  18 186  20   4   2   2   2   0  51]
 [ 33   6  19 172   4   3  21  10   2  20]
 [  2   2   0   2 231  33   0   8   7   5]
 [ 27   3   1   8   8 181  12  35  13   2]
 [ 62   0   0  14   0   6 194  14   0   0]
 [ 52   2   2  11   5  40  14 160   3   0]
 [  2  10   0   7  12  28   1  12 217   1]
 [  1  17  35  14   6   2   0   3   0 212]]
........................................Epoch: 51/200,  Loss: 0.0196
........................................Epoch: 51/200,  Loss: 0.0208
........................................Epoch: 51/200,  Loss: 0.0223
........................................Epoch: 51/200,  Loss: 0.0147
........................................Epoch: 51/200,  Loss: 0.0144
........................................Epoch: 51/200,  Loss: 0.0132
.............................           Epoch 51 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 70.1482 %
Confusion matrix
[[219   0   0  14   4   7  32  12   0   3]
 [  1 259   3   7   0   0   2   0   2  16]
 [  3  17 156  37   0   1   5   2   0  70]
 [ 17   5  16 186   3   1  34   5   2  21]
 [  3   0   1   2 245  18   0   6   9   6]
 [ 40   1   2   8  20 153  16  36  13   1]
 [ 45   0   0  16   0  10 210   9   0   0]
 [ 53   0   3  19   4  25  23 158   4   0]
 [  1   4   0  13  17  23   1   9 221   1]
 [  2  12  19  21   5   1   0   2   0 228]]
........................................Epoch: 52/200,  Loss: 0.0179
........................................Epoch: 52/200,  Loss: 0.0221
........................................Epoch: 52/200,  Loss: 0.0149
........................................Epoch: 52/200,  Loss: 0.0194
........................................Epoch: 52/200,  Loss: 0.0254
........................................Epoch: 52/200,  Loss: 0.0160
.............................           Epoch 52 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.7349 %
Confusion matrix
[[235   0   1   3   5   5  24  16   1   1]
 [  0 267   4   4   3   0   1   1   3   7]
 [  8  22 162  39   4   4   3   5   3  41]
 [ 28   7  13 173   7   4  35  11   4   8]
 [  2   1   0   2 251  21   0   6   6   1]
 [ 46   1   0   8  23 134  16  47  15   0]
 [ 61   0   0  14   0   7 197  11   0   0]
 [ 54   1   2   7   4  19  29 170   3   0]
 [  2   6   0   8  23  17   0   9 225   0]
 [  4  28  37  22  11   1   2   3   2 180]]
........................................Epoch: 53/200,  Loss: 0.0193
........................................Epoch: 53/200,  Loss: 0.0209
........................................Epoch: 53/200,  Loss: 0.0228
........................................Epoch: 53/200,  Loss: 0.0180
........................................Epoch: 53/200,  Loss: 0.0158
........................................Epoch: 53/200,  Loss: 0.0234
.............................           Epoch 53 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 70.2172 %
Confusion matrix
[[238   0   1   5   4  12  13  15   1   2]
 [  1 249  12   5   3   0   1   2   3  14]
 [  4   8 211  18   2   4   4   3   2  35]
 [ 26   4  31 157   6   5  31  17   6   7]
 [  0   0   0   1 244  21   0   6  13   5]
 [ 25   0   1   5  13 153  12  44  36   1]
 [ 73   1   1   8   0  10 173  23   1   0]
 [ 44   0   2   8   5  33  10 182   5   0]
 [  1   2   0   5  11  14   1   5 249   2]
 [  2   9  64  18   8   1   0   5   2 181]]
........................................Epoch: 54/200,  Loss: 0.0259
........................................Epoch: 54/200,  Loss: 0.0257
........................................Epoch: 54/200,  Loss: 0.0247
........................................Epoch: 54/200,  Loss: 0.0261
........................................Epoch: 54/200,  Loss: 0.0201
........................................Epoch: 54/200,  Loss: 0.0166
.............................           Epoch 54 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.6660 %
Confusion matrix
[[192   0   1   7   3  23  42  20   0   3]
 [  1 250   5   6   1   0   1   4   1  21]
 [  2  11 167  25   0   7   5   2   2  70]
 [ 20   4  25 162   7  10  32  13   2  15]
 [  0   0   1   2 244  23   0   5   9   6]
 [ 21   0   2   1  19 201  12  22  12   0]
 [ 25   1   0  10   0  19 216  19   0   0]
 [ 32   0   3   8   4  88  23 127   4   0]
 [  3   4   0   7  18  32   1   5 218   2]
 [  2   7  30  26   5   2   0   2   1 215]]
........................................Epoch: 55/200,  Loss: 0.0282
........................................Epoch: 55/200,  Loss: 0.0174
........................................Epoch: 55/200,  Loss: 0.0226
........................................Epoch: 55/200,  Loss: 0.0285
........................................Epoch: 55/200,  Loss: 0.0147
........................................Epoch: 55/200,  Loss: 0.0249
.............................           Epoch 55 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 69.5967 %
Confusion matrix
[[225   0   0   1   0  20  16  25   1   3]
 [  0 259  10   2   0   0   0   1   3  15]
 [  5  10 213  12   0   8   1   1   3  38]
 [ 20   4  47 149   4   7  32  18   3   6]
 [  0   0   0   3 190  67   0  11  15   4]
 [ 20   1   3   1   2 197  13  32  21   0]
 [ 62   1   0  11   0  15 184  17   0   0]
 [ 28   0   2   9   1  49  16 180   4   0]
 [  0   3   0   7   3  29   0   7 239   2]
 [  2   8  69  17   4   1   0   4   2 183]]
........................................Epoch: 56/200,  Loss: 0.0136
........................................Epoch: 56/200,  Loss: 0.0191
........................................Epoch: 56/200,  Loss: 0.0206
........................................Epoch: 56/200,  Loss: 0.0159
........................................Epoch: 56/200,  Loss: 0.0196
........................................Epoch: 56/200,  Loss: 0.0211
.............................           Epoch 56 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 66.8735 %
Confusion matrix
[[226   0   1   1   3  17   6  33   1   3]
 [  0 270   3   1   1   0   1   0   4  10]
 [ 12  22 170  12   3   9   4   4   4  51]
 [ 46   5  24 114   8  22  32  20   6  13]
 [  2   0   1   0 238  28   0   7  11   3]
 [ 32   1   2   1  17 172   7  50   8   0]
 [ 81   1   0   5   0   7 161  34   1   0]
 [ 35   0   1   1   4  51   9 184   4   0]
 [  1   3   0   3  17  28   0  10 226   2]
 [  7  32  45   6   6   5   0   8   2 179]]
........................................Epoch: 57/200,  Loss: 0.0170
........................................Epoch: 57/200,  Loss: 0.0178
........................................Epoch: 57/200,  Loss: 0.0195
........................................Epoch: 57/200,  Loss: 0.0146
........................................Epoch: 57/200,  Loss: 0.0163
........................................Epoch: 57/200,  Loss: 0.0183
.............................           Epoch 57 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.4936 %
Confusion matrix
[[186   0   0   2   2  18  24  55   1   3]
 [  0 263   1   2   2   0   3   0   3  16]
 [  3  18 172  16   1   8   6   4   4  59]
 [ 21   5  20 136   4  18  43  25   3  15]
 [  0   0   1   0 220  45   0   9  11   4]
 [ 17   1   1   1  10 172  15  56  17   0]
 [ 42   0   0   2   0  12 202  31   1   0]
 [ 17   0   1   6   2  35  15 209   4   0]
 [  1   3   0   4   9  30   0  12 228   3]
 [  3  13  44  12   5   6   0   6   2 199]]
........................................Epoch: 58/200,  Loss: 0.0229
........................................Epoch: 58/200,  Loss: 0.0139
........................................Epoch: 58/200,  Loss: 0.0182
........................................Epoch: 58/200,  Loss: 0.0137
........................................Epoch: 58/200,  Loss: 0.0172
........................................Epoch: 58/200,  Loss: 0.0159
.............................           Epoch 58 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.4936 %
Confusion matrix
[[189   0   0   5   8  25  26  34   2   2]
 [  0 255   2   2   5   1   1   1   2  21]
 [  3  20 161  21  16   7   4   3   3  53]
 [ 15   6  18 159  16  15  26  19   2  14]
 [  0   0   0   2 268  11   0   2   6   1]
 [ 17   1   0   3  48 174   7  25  15   0]
 [ 38   1   0   8   2  21 188  31   1   0]
 [ 21   0   1   5  22  56  10 170   4   0]
 [  1   3   0   6  31  27   0   1 220   1]
 [  2   9  36  20   9   4   1   4   2 203]]
........................................Epoch: 59/200,  Loss: 0.0121
........................................Epoch: 59/200,  Loss: 0.0258
........................................Epoch: 59/200,  Loss: 0.0156
........................................Epoch: 59/200,  Loss: 0.0161
........................................Epoch: 59/200,  Loss: 0.0254
........................................Epoch: 59/200,  Loss: 0.0246
.............................           Epoch 59 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 67.8731 %
Confusion matrix
[[236   0   1   5   2  12  16  16   1   2]
 [  1 239  18  14   1   0   5   2   3   7]
 [  8   7 211  22   9   4   4   5   0  21]
 [ 40   3  27 165   5   3  23  13   2   9]
 [  5   0   1   4 244  25   0   4   7   0]
 [ 36   1   1   4  16 173   9  39  11   0]
 [ 72   0   1   8   0  16 169  23   1   0]
 [ 49   0   2   9   6  37  11 171   4   0]
 [  2   2   0   7  18  27   0  17 216   1]
 [  4  11  85  28   9   2   0   4   2 145]]
........................................Epoch: 60/200,  Loss: 0.0120
........................................Epoch: 60/200,  Loss: 0.0159
........................................Epoch: 60/200,  Loss: 0.0203
........................................Epoch: 60/200,  Loss: 0.0163
........................................Epoch: 60/200,  Loss: 0.0122
........................................Epoch: 60/200,  Loss: 0.0211
.............................           Epoch 60 completed
----------------------------------------
----------------------------------------
-----------
Accuracy: 68.6660 %
Confusion matrix
[[188   0   0   3   4  17  38  36   2   3]
 [  0 240   3  16   2   1   1   4   3  20]
 [  5  13 168  32   0   3   1   9   3  57]
 [ 26   5  15 162   5   6  31  19   5  16]
 [  2   0   2   3 230  25   0   8  15   5]
 [ 21   0   0   5   9 153  12  58  32   0]
 [ 29   0   0  11   0   9 206  34   1   0]
 [ 25   0   1   6   3  35  18 195   6   0]
 [  1   4   0   7   7  15   0  12 243   1]
 [  3  10  36  21   3   1   1   6   2 207]]
No improvement for 10 epochs

Achieved accuracy: 70.52740434332989

Evaluation

The best fitting model has been saved as file checkpoint.pt. Load it and check the accuracy against the test data set.

In [25]:
# To avoid disk wear the checkpoints have been saved in the /tmp directory
# which hopefully is a RAM disk. Copy the current checkpoint to our directory.
os.popen("cp /tmp/checkpoint.pt checkpoint.pt 2>&1 && echo ok").read()
Out[25]:
'ok\n'
In [26]:
def load_checkpoint(path='checkpoint.pt'):
    """Reload model from checkpoint
    """
    checkpoint = torch.load(path)
    model = checkpoint['model']
    return model
In [27]:
# Check the accuary using the test data
model = load_checkpoint('checkpoint.pt')
class_to_idx = model.class_to_idx
check_accuracy(model, test_loader, mute=False)
pass
----------------------------------------
----------------------------------------
-----------
Accuracy: 69.2892 %
Confusion matrix
[[227   0   4  10   1  14  28   6   0   0]
 [  0 261  10   3   0   0   0   3   0  13]
 [  3  14 165  31   2   1   2   2   0  69]
 [ 26   6  28 167   2  10  19  13   3  16]
 [  1   0   1   3 249  20   0   3   7   6]
 [ 30   1   4   5  15 165  18  33  16   3]
 [ 58   0   1  13   0   7 198  12   0   1]
 [ 54   2   2   7   4  42  20 149   6   3]
 [  0   4   2   5  12  38   1  11 217   0]
 [  2  21  31  16   2   3   0   5   0 210]]

The columns in the confusion matrix correspond to the predicted categories. The rows to the actual categories. Find the labels for the rows an columns below.

So a the noise inside a bus is easily recognized. But the sound of a street with pedestrians is easily mistake for a public square.

In [28]:
class_to_idx
Out[28]:
{'airport': 0,
 'bus': 1,
 'metro': 2,
 'metro_station': 3,
 'park': 4,
 'public_square': 5,
 'shopping_mall': 6,
 'street_pedestrian': 7,
 'street_traffic': 8,
 'tram': 9}

Prediction

Let's use the model to make a prediction

In [29]:
class Predict:
    """Image classifier"""

    def __init__(self):
        """Constructor"""
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        self.image = None
        self.category_names = None
        self.top_k = 5

    def classify(self, image_file_name, checkpoint_file_name,
                 category_names_file_name = None):
        """classify an image"""
        self.load_checkpoint(checkpoint_file_name)
        self.load_image(image_file_name)
        if category_names_file_name is not None:
            self.load_category_names(category_names_file_name)
        probs, categories = self.infer(self.top_k)
        self.output(probs, categories)
        self.draw_prediction(probs, categories)

    def infer(self, top_k):
        """Infer the classes"""
        self.model.to(self.device)
        self.model.eval()
        inputs = torch.stack((self.image,))
        inputs = inputs.to(self.device)

        with torch.no_grad():
            outputs = self.model(inputs)

        outputs = outputs.to("cpu")
        probs, indices = outputs.topk(top_k)
        probs = probs.exp()
        probs = probs.tolist()[0]
        indices = indices.tolist()[0]
        categories = [self.idx_to_class[index] for index in indices]
        return probs, categories

    def load_checkpoint(self, checkpoint_file_name):
        """Load checkpoint from file"""
        checkpoint = torch.load(checkpoint_file_name, map_location={'cuda:0': 'cpu'})
        self.model = checkpoint['model']
        self.criterion = nn.NLLLoss()
        class_to_idx = self.model.class_to_idx
        self.idx_to_class = {value : key
                             for key, value in class_to_idx.items()}

    def load_image(self, image_file_name):
        """Load image from file"""
        self.image = Image.open(image_file_name).convert('RGB')
        self.image = self.normalize_image(self.image)

    def normalize_image(self, image):
        """Normalize image"""
        return test_transforms(image)

    def output(self, probs, categories):
        """Output category names and propabilities"""
        if self.category_names is not None:
            categories = [self.category_names[category]
                          for category in categories]
            category_title = 'Category Name'
        else:
            category_title = 'Category'
        max_len = max([len(category) for category in categories])
        max_len = max(max_len, len(category_title))
        print('{:>{}} | {}'.format(category_title, max_len, 'Propability'))
        print('{:>{}}-+-{}'.format('-' * max_len, max_len, '-----------'))
        for i in range(len(probs)):
            print('{:>{}} | {:.4f}'.format( categories[i], max_len, probs[i]))

    def draw_prediction(self, propabilities, categories,
                        true_category = '', title = 'Prediction'):
        """Draw a plot showing the prediction"""

        fig, plot = plt.subplots(nrows=1, ncols=1, figsize=(7,5))
        fig.tight_layout()
        fig.suptitle(title, fontsize=20, y=1.05)

        margin = 0.05
        n_cat = len(categories)
        ind = np.arange(n_cat)
        width = (1. - 2. * margin) / n_cat
    
        plot.barh(ind, propabilities[::-1], height = .8)
        plot.set_yticks(ind + margin)
        plot.set_yticklabels(categories[::-1], fontsize=16)
        plot.set_xlabel('propability')
            
    def set_device(self, device_name):
        """Set the cuda device"""
        self.device = torch.device(device_name)

    def set_top_k(self, top_k):
        """Set number of categories to output"""
        self.top_k = top_k
        
In [30]:
# Can you identify the sound?
x = readAudioFile(rawDataPath + 'bus-milan-1115-42136-a.wav')
ipd.Audio(x, rate=sr)
Out[30]:
In [31]:
# Check your guess against the prediction.
predict = Predict()
predict.classify('data/spectrograms/bus/milan/1115-42136-a.png', 'checkpoint.pt', None)
     Category | Propability
--------------+------------
          bus | 0.9268
metro_station | 0.0350
        metro | 0.0335
shopping_mall | 0.0029
         tram | 0.0018

Refinement

The parameters chosen for the model above provide a reasonable accuracy. But can we do better?

With limited project time let's just concentrate on two of them controlling augmentation.

In [32]:
def refine():
    """Optimize hyper parameters"""
    
    # If we have enough computation time, we should use multiple runs with different
    # random seeds per run to get a more reliable measure for the accuracy.
    repetitions = 3

    # To make this reproducible let'use fixed random seeds
    seeds = [8259, 6745, 14986, 4215];

    best_accuracy = 0
    best_parameters = {}
    best_testloader = None
    best_test_transforms = None
    
    sample_sizes = [.3, .5, .7]
    max_widths = [0., .1, .2]

    for sample_size in sample_sizes:
        for max_width in max_widths:
            accuracies = []
            for i in range(repetitions):
                torch.manual_seed(seeds[i])
                random.seed(seeds[i])
                accuracy, model, test_loader, test_transforms, class_to_idx = \
                         run_training(sample_size=sample_size, max_width=max_width,
                                      hidden_units=512, epochs=200, mute=True)
                accuracies.append(accuracy)
            average_accuracy = statistics.mean(accuracies)
            if (average_accuracy > best_accuracy):
                best_accuracy = average_accuracy
                best_parameters= {
                    'sample_size' : sample_size,
                    'max_width' : max_width
                }
                os.popen("cp /tmp/checkpoint.pt /tmp/best_checkpoint.pt 2>&1")
                print('new best');
            print('sample_size = {}, max_width = {}, average_accuracy = {:.4f} +/- {:.4f} %'.format(
                sample_size, max_width, average_accuracy, statistics.stdev(accuracies)))

    return best_parameters, best_accuracy

best_parameters, best_accuracy = refine()    
print('best parameters: {}'.format(best_parameters))
print('best accuracy: {:.4f} %'.format(best_accuracy))
os.popen("cp /tmp/best_checkpoint.pt best_checkpoint.pt 2>&1")
pass
********************************
*************************
***********************
new best
sample_size = 0.3, max_width = 0.0, average_accuracy = 60.2436 +/- 0.8275 %
*************************************
**********************************
**************************************
sample_size = 0.3, max_width = 0.1, average_accuracy = 58.6694 +/- 0.2260 %
*********************************************
************************************
************************
sample_size = 0.3, max_width = 0.2, average_accuracy = 58.1868 +/- 0.6233 %
**************************************
*****************************************************
*************************************************************
new best
sample_size = 0.5, max_width = 0.0, average_accuracy = 71.2168 +/- 0.7623 %
******************************************************
*************************************************
*************************************************
sample_size = 0.5, max_width = 0.1, average_accuracy = 70.7687 +/- 0.0345 %
***************************************
*********************************************
**************************************************
sample_size = 0.5, max_width = 0.2, average_accuracy = 69.5277 +/- 0.7607 %
******************
*********************
******************************************
sample_size = 0.7, max_width = 0.0, average_accuracy = 63.5988 +/- 1.7163 %
****************
*********************
*****************************
sample_size = 0.7, max_width = 0.1, average_accuracy = 62.9323 +/- 2.4239 %
***************************************
****************************************
*************************
sample_size = 0.7, max_width = 0.2, average_accuracy = 64.5754 +/- 0.6651 %
best parameters: {'sample_size': 0.5, 'max_width': 0.0}
best accuracy: 71.2168 %
In [34]:
# Check the accuary using the test data
model = load_checkpoint('best_checkpoint.pt')
class_to_idx = model.class_to_idx
check_accuracy(model, test_loader, mute=False)
pass
----------------------------------------
----------------------------------------
-----------
Accuracy: 69.6687 %
Confusion matrix
[[217   0   3   3   0  22  34  11   0   0]
 [  1 255  21   0   0   0   0   1   2  10]
 [  6  15 202  21   1   2   0   1   0  41]
 [ 33   4  39 158   0  15  14  11   5  11]
 [  4   0   0   0 244  27   0   6   6   3]
 [ 18   1   1   6  13 182  12  42  15   0]
 [ 37   0   3   9   0  10 210  19   1   1]
 [ 35   1   2   9   5  58  19 152   6   2]
 [  2   1   0   4  14  48   1  10 209   1]
 [  4  21  54  13   3   1   0   3   1 190]]

The accuracy is improved slightly. But given the standard deviations shown in the grid search the difference is within the error margins of accuracy.

The grid search results indicate for the different augmentation techniques applied:

  • using a random 5 second interval out of the 10 second recordings is a reasonalble setting for time warping
  • frequency band masking has a negative effect on the accuracy.

Deployment

The project folders contains two standalone Python programs.

  • train.py can be used to retrain the model on a new data set.
  • predict.py uses a saved checkpoint for predictions. Similar code could be used to create a web service for classification.

Conclusion

A workflow for classifying ambiant sound was demonstrated:

  • correcting sound quality by using a pre-emphasis filter
  • convert sound files to spectrograms
  • split the dataset into training, validation, and testing data
  • train a convolutional neural network using data augmentation
  • use the trained network for prediction

Though a network was used that is not specificically built for this classification task respectable accuracy rates were be achieved.

Directions for further investigation could be

  • a grid search to optimize the parameters of the augmentation transforms
  • testing further augmentation techniques like adding noise
  • design of a more specific neural network.